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DETAILED ACTION 
Response to Amendment 
1. The amendment filed on 01/20/2004 has been entered. Claims 1, 2, 6-7, and 1 1-13 
have been amended. Claims 26-28 have been newly added. Claims 3-4; 9-10; 14-25 have 
been canceled. 



Claim Rejections - 35 USC §103 

2. The following is a quotation of 35 U.S. C 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-2, 5-8, 1 1-13, and 26-28 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Xiong et al. U.S. Patent No. 6,434,265 (hereinafter Xiong), and fiirther 
in view of Shum et al. U.S. Patent No. 6,271,855 (hereinafter Shum) and Teo U.S. Patent 
No. 6,246,413 (Teo). 

4. Claim 1: 

(1) Xiong teaches an image synthesis method comprising: 

An input step, of inputting a plurality of image data (e.g., column 9, lines 20-30); 

A placement information generating step, of generating placement information 
determined by a placement order of all images inputted in the input step (e.g., column 16, 
lines 35-45); 
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A placement information obtaining step of obtaining placement information about 
a plurality of images in which adjacent images have a common subject region (e.g., in 
column 4, lines 5-40, Xiong teaches a method for constructing a panorama from 
rectilinear images in 3D through projective registration and calibration including: (1) the 
projective registrations of overlapping images, (2) calibration and global optimization of 
these images, a self calibration in which 2D image planes are positioned as 3D planes in 
space); 

A setting step of setting one mapping mode out of a plurality of mapping modes 
each corresponding to a different mapping surface in accordance with the obtained 
placement information (e.g., in column 8, lines 18-58 of Xiong, it is stated that 
"overlapping photographs are analyzed to determine what orientation the photographs 
were taken in order to establish a common ground for subsequent operations and the 
panorama is constructed on a particular geometry that vnW best facilitate the rendering of 
the projection of the panorama onto a chosen viewing plane for viewing". The Xiong 
discloses some typical geometry on which panoramas are formed; In column 8, lines 18- 
58, Xiong flirther discloses that panorama is constructed on a particular geometry that 
will best facilitate the rendering of the projection onto a chosen viewing plane such as 
cubic, polyhedral, cylindrical and spherical geometries); and 

A synthesis step of combining the plurality of images by using the mapping mode 
set in the setting step (e.g., in column 4, lines 5-40 of Xiong, it is stated that the 
composing or blending in which images are ready to be re-projected to a 3D environment 
map with pixels in overlap regions being composed from muhiple; In column 8, lines 18- 
58, Xiong further teaches that overlapping photographs are analyzed to determine what 
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orientation the photographs were taken in order to establish a common ground for 
subsequent operations and the panorama is constructed on a particular geometry that will 
best facilitate the rendering of the projection of the panorama onto a chosen viewing 
plane for viewing. The Xiong further discloses some typical geometry on which 
panoramas are formed); 

A changing step, of changing the mapping mode (In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a changing step through the user interface 230 by selecting a geometric surface 
out of a plurality of geometric surfaces each corresponding to a different mapping 
surface). 



- The Examiner interprets "a placement information obtaining step of obtaining 
placement information about a plurality of images in which adjacent images have 
a common subject region" as an automatic registration and calibration step of 
registering the overlapping images and capturing common overlapping areas 
between overlapping images and minimizing the average squared pixel intensity 
difference with respect to certain transformation parameters. 

- The Examiner interprets the mapping mode as mapping images onto a geometric 
surface such as a planar or a cylindrical surface (Applicant's specification, page 1, 
lines 20-25), Accordingly, the Examiner interprets the setting step of setting a 
mapping mode as the selecting step of selecting a geometric surface. In column 8, 
lines 18-58, Xiong discloses that panorama is constructed on a particular 
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geometry that will best facilitate the rendering of the projection onto a chosen 
viewing plane (a geometric surface) such as cubic, polyhedral, cylindrical and 
spherical geometries. In column 4, lines 40-50, it is stated that "the projection 
module may be controlled through the user interface 230 as well, to allow a user 
to select what geometry will be projected onto". Therefore, Xiong teaches a 
selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong 
fulfills the claimed limitation of a setting step of setting one mapping mode out of 
a plurality of mapping modes each corresponding to a different mapping surface 
in accordance with said obtained placement information. 

(2) However, Xiong does not implicitly teach the claimed limitation of "a 
generating step, of issuing , when an image formed by changing the mapping mode in the 
changing step does not comply with a predetermined condition set in accordance with the 
mapping mode, a warning and generating a synthesized image in accordance with the 
predetermined condition" in its totality (i.e., generation of a warning message). Xiong does 
not implicitly teach "a setting step, of automatically setting one mapping mode out of a 
plurality of mapping modes each corresponding to a different mapping surface in 
accordance with the obtained placement information." 

(3) Shum implicitly teaches the claimed limitation of a generating step, of 
issuing , when an image formed by changing the mapping mode in said changing step 
does not comply with a predetermined condition set in accordance with the mapping 
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mode, a warning and generating a synthesized image in accordance with the 
predetermined condition (Shum abstract; column 18, lines 55-65). 

Teo teaches the claim limitation of a setting step, of automatically setting one 
mapping mode out of a plurality of mapping modes each corresponding to a different 
mapping surface in accordance with the obtained placement information (Teo teaches 
manipulating (adjusting the scale of the polyhedral surface, rotating the surface, re- 
positioning edges of the polyhedral surfaces) a polyhedral surface upon which the scene 
is to be projected, relative to the initial panoramic image, to form a desired surface, 
wherein the desired surface is distinct from the initial surface and modifying the initial 
panoramic image by mapping points on the desired surface to corresponding points on 
the initial surface to produce a modified panoramic image corresponding to projection of 
the scene onto the desired surface. Teo therefore teaches automatically setting a mapping 
mode by the computer through a user interface); 

(4) It would have been obvious to one of ordinary skill in the art to have 
incorporated the Shum's warning message generation step because Xiong suggests a 
generating step of generating a synthesized image in accordance with the predetermined 
condition (Xiong column 3, lines 35-55; column 17, lines 15-67; column 18, lines 1-4). 
Moreover, Xiong teaches in figures 2 and 3 a user interface and a global optimization that 
provides feedback to the computer system such as issuing warning messages on the 
computer monitor 218 when the pair-wise objective function is not desirable for a poor 
selection of the projection viewing plane (and the resulting panoramas will have 
imperfectly aligned images that give shadow or ghosting effects) due to a poor selection 
of the projection viewing plane and the resuhing panoramas will have imperfectly aligned 
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images that give shadow or ghosting effects. Xiong also points to a seamless multi- 
resolution average blending method that would result in an absent of shadow effects 
(column 14, lines 1-45). Xiong further teaches a human interaction being present 
throughout the image synthesis process to provide feedback to the computer system in all 
the nonlinear optimizations to let users monitor the progress of the system and allow 
them to intervene when necessary and therefore the claimed limitation suggests an 
obvious modification of Xiong because providing feedback to users is similar to issuing a 
message to users in a user interaction with the nonlinear optimizations when the changing 
step does not comply with a predetermined condition set or user-selected parameter set. 

Furthermore, Xiong also teaches a user interface for setting a mapping surface by 
the computer and therefore Xiong suggests automatically setting a mapping mode. 

(5) One having the ordinary skill in the art would have been motivated to do this 
because this would have provided the user the informative message such as issuing a 
warning message through the user interaction when convergence to the predetermined 
condition setting cannot be obtained in the image synthesis (Xiong column 17, lines 15- 
67, column 18, lines 1-4) for non-solvable optimization problems (Shum column 18, lines 
38-67). One having the ordinary skill in the art would have been motivated to do this 
because setting a mapping mode out from a plurality of mapping mode serves for the 
purpose of eliminating visible artifacts (Teo column 10). 

Claim 2: 

Claim 2 recites all the limitations of claim 1 and adds the limitation of "a focal 
length obtaining step." Xiong teaches finding projective parameters such as 3D rotation 
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parameters (pan, tilt roll), center of projection of images, ratio of focal lengths, and the 
like (column 10, lines 9-28). Xiong implicitly teaches finding the camera internal 
parameters vector of which the focal length is a component (column 1 1, lines 15-42). 



Claim 5: 

The claim 5 recites all the limitations of claim 1 and adds the limitation of "a 
displaying step of displaying a cuttable rectangular region." Xiong teaches how to align 
images more precisely by changing the coordinates for positioning an image. Xiong 
further teaches placing the images 1210 at selected tangents on the viewing sphere 1220 
(figure 12, and column 17, Hnes 12-65). 



5. Claim 6: 

The claim 6 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of "an image synthesis apparatus" and "a displaying step of 
displaying a cuttable rectangular region". However, Xiong further discloses in figure 3 an 
apparatus for image synthesis. Xiong teaches how to align images more precisely by 
changing the coordinates for positioning an image. Xiong further teaches placing the 
images 1210 at selected tangents on the viewing sphere 1220 (figure 12, and column 17, 
lines 12-65). 

Claim 7: 

The claim 7 recites all the limitations of claim 6 and adds the limitation of "a 
focal length obtaining step." The Xiong reference teaches finding projective parameters 
such as 3D rotation parameters (pan, tilt roll), center of projection of images, ratio of 
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focal lengths, and the like (column 10, lines 9-28). Xiong implicitly teaches finding the 
camera internal parameters vector in his image synthesis apparatus of figure 3 because 
the focal length is a component of that vector (column 1 1, lines 15-42). 
Claim 8: 

The claim 8 recites all the limitations of claim 6 and adds the limitation of "a 
changing step of changing the mapping mode." Xiong teaches that panorama is 
constructed on a particular geometry that will best facilitate the rendering of the 
projection of the panorama onto a chosen viewing plane such as cubic, polyhedral, 
cylindrical and spherical geometries (column 8, lines 18-58), 



6. Claim 11: 

The claim 1 1 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program 
for implementing image synthesis method. However, Xiong further discloses the claimed 
limitation of a computer-readable storage medium having a program for implementing 
image synthesis method (e.g., in column 3, lines 54-57, it is stated "a program residing in 
system memory 220 which stores output data and other data"). 

Claim 12: 

The claim 12 recites all the limitations of claim 1 1 and adds "a focal length 
obtaining step." Xiong teaches finding projective parameters such as 3D rotation 
parameters (pan, tih roll), center of projection of images, ratio of focal lengths, and the 
like (column 10, lines 9-28). Xiong implicitly teaches finding the camera internal 
parameters vector of which the focal length is a component (column 11, lines 15-42). 
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Claim 13: 

The claim 13 recites all the limitations of claim 1 1 and adds the limitation of "a 
changing step of changing the mapping mode." Xiong teaches that panorama is 
constructed on a particular geometry that will best facilitate the rendering of the 
projection of the panorama onto a chosen viewing plane such as one of the cubic, 
polyhedral, cylindrical and spherical geometries (column 8, lines 18-58). 



7. Claim 26: 

(1) Xiong teaches an image synthesis method comprising: 

An input step, of inputting a plurality of image data (e.g., column 9, lines 20-30); 

A placement information generating step, of generating placement information 
determined by a placement order of all images inputted in the input step (e.g., column 16, 
lines 35-45); 

A placement information obtaining step of obtaining placement information about 
a plurality of images in which adjacent images have a common subject region (e.g., in 
column 4, lines 5-40, Xiong teaches a method for constructing a panorama from 
rectilinear images in 3D through projective registration and calibration including: (1) the 
projective registrations of overlapping images, (2) calibration and global optimization of 
these images, a self calibration in which 2D image planes are positioned as 3D planes in 
space); 

A setting step of setting one mapping mode out of a plurality of mapping modes 
each corresponding to a different mapping surface in accordance with the obtained 
placement information (e.g., in column 8, lines 18-58 of Xiong, it is stated that 
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"overlapping photographs are analyzed to determine what orientation the photographs 
were taken in order to establish a common ground for subsequent operations and the 
panorama is constructed on a particular geometry that will best facilitate the rendering of 
the projection of the panorama onto a chosen viewing plane for viewing". The Xiong 
discloses some typical geometry on which panoramas are formed; In column 8, lines 18- 
58, Xiong further discloses that panorama is constructed on a particular geometry that 
will best facilitate the rendering of the projection onto a chosen viewing plane such as 
cubic, polyhedral, cylindrical and spherical geometries); and 

A synthesis step of combining the plurality of images by using the mapping mode 
set in the setting step (e.g., in column 4, lines 5-40 of Xiong, it is stated that the 
composing or blending in which images are ready to be re-projected to a 3D environment 
map with pixels in overlap regions being composed from multiple; In column 8, lines 18- 
58, Xiong further teaches that overlapping photographs are analyzed to determine what 
orientation the photographs were taken in order to establish a common ground for 
subsequent operations and the panorama is constructed on a particular geometry that will 
best facilitate the rendering of the projection of the panorama onto a chosen viewing 
plane for viewing. The Xiong further discloses some typical geometry on which 
panoramas are formed); 



The Examiner interprets "a placement information obtaining step of obtaining 
placement information about a plurality of images in which adjacent images have 
a common subject region" as an automatic registration and calibration step of 
registering the overlapping images and capturing common overlapping areas 
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between overlapping images and minimizing the average squared pixel intensity 
difference with respect to certain transformation parameters. 
- The Examiner interprets the mapping mode as mapping images onto a geometric 
surface such as a planar or a cylindrical surface (Applicant's specification, page 1, 
lines 20-25). Accordingly, the Examiner interprets the setting step of setting a 
mapping mode as the selecting step of selecting a geometric surface. In column 8, 
lines 18-58, Xiong discloses that panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection onto a chosen 
viewing plane (a geometric surface) such as cubic, polyhedral, cylindrical and 
spherical geometries. In column 4, lines 40-50, it is stated that "the projection 
module may be controlled through the user interface 230 as well, to allow a user 
to select what geometry will be projected onto". Therefore, Xiong teaches a 
selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong 
fulfills the claimed limitation of a setting step of setting one mapping mode out of 
a plurality of mapping modes each corresponding to a different mapping surface 
in accordance with said obtained placement information. 

(2) However, Xiong does not implicitly teach "a setting step, of automatically 
setting one mapping mode out of a plurality of mapping modes each corresponding to a 
different mapping surface in accordance with the obtained placement information." 

(3) Teo teaches the claim limitation of a setting step, of automatically setting one 
mapping mode out of a plurality of mapping modes each corresponding to a different 
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mapping surface in accordance with the obtained placement information (Teo teaches 
manipulating (adjusting the scale of the polyhedral surface, rotating the surface, re- 
positioning edges of the polyhedral surfaces) a polyhedral surface upon which the scene 
is to be projected, relative to the initial panoramic image, to form a desired surface, 
wherein the desired surface is distinct from the initial surface and modifying the initial 
panoramic image by mapping points on the desired surface to corresponding points on 
the initial surface to produce a modified panoramic image corresponding to projection of 
the scene onto the desired surface. Teo therefore teaches automatically setting a mapping 
mode by the computer through a user interface); 

(4) It would have been obvious to one of ordinary skill in the art to have 
incorporated the Teo's automatically setting step because Xiong teaches a user interface 
for setting a mapping surface by the computer automatically and therefore Xiong 
suggests automatically setting a mapping mode. 

(5) One having the ordinary skill in the art would have been motivated to do this 
because setting a mapping mode out from a plurality of mapping mode serves for the 
purpose of eliminating visible artifacts (Teo column 10). 



8. Claim 27: 

The claim 27 encompasses the same scope of invention as that of the claim 26 
except additional claim limitation of "a displaying step of displaying a cuttable 
rectangular region." Xiong teaches how to align images more precisely by changing the 
coordinates for positioning an image. Xiong further teaches placing the images 1210 at 
selected tangents on the viewing sphere 1220 (figure 12, and column 17, lines 12-65). 
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9. Claim 28: 

The claim 28 encompasses the same scope of invention as that of claim 26 except 
additional claimed limitation of a computer-readable storage medium having a program 
for implementing image synthesis method. However, Xiong further discloses the claimed 
limitation of a computer-readable storage medium having a program for implementing 
image synthesis method (e.g., in column 3, lines 54-57, it is stated "a program residing in 
system memory 220 which stores output data and other data"). 

Remarks 

10. Applicant's arguments, filed 01/20/2004, paper number 12, have been fully 
considered but they are not deemed to be persuasive. 

11. Applicant argues in essence with respect to the amended claim 1 and similar 
claims that: 

"Furthermore in Xiong et al., each image is synthesized onto a predetermined 
surface one by one to determine its placement (Xiong et al, col. 15, lines 20-33). 
Such a step would not enable the method disclosed by Xiong et al. to determine 
placement information that is determined by the placement order of all images, as 
taught in Claim 1 of the present invention." 
This is not found persuasive for the following reasons: 

Xiong teaches authoring of the panorama to allow manual input to override the 
computer defauh ordering (placement order of all images) for the blending of the 
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images. Images with larger overlaps should be blended onto the panoramic canvas 
first manually, and can be selected for blending by a user of the present apparatus, 
through a suitable user interface that lists the images to be blended first in a 
priority list (placement order of all images). Xiong therefore teaches determining 
placement information that is determined by the placement order of all images as 
claimed in Claim 1. 



12. Applicant argues in essence with respect to the amended claim 1 and similar 
claims that: 

''Xiong et al is not seen to teach or suggest how to select among plural surfaces, 
as in Claim 1. In addition, the Office Action at page 4 stages that, according to 
col 4, lines 40-50 of Xiong et al, a user can manually select (through a user 
interface 230) a geometry which will be projected onto. However, Xiong et al. is 
not seen to teach or suggest automatically setting one mapping mode out of a 
plurality of mapping modes each corresponding to a different mapping surface in 
accordance with the obtained placement information, as recited in Claim 1." 
This is not found persuasive because of the following reasons: 

In column 8, lines 18-58, Xiong discloses that panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen 
viewing plane (a geometric surface) such as cubic, polyhedral, cylindrical and spherical 
geometries. In column 4, lines 40-50, it is stated that "the projection module may be 
controlled through the user interface 230 as well, to allow a user to select what geometry 
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will be projected onto". Therefore, Xiong teaches a selecting step of selecting a 
geometric surface out of a plurality of geometric surfaces each corresponding to a 
different mapping surface in accordance with the obtained placement information. As 
applied to the present application, Xiong flilfills the claimed limitation of a setting step of 
setting one mapping mode out of a plurality of mapping modes each corresponding to a 
different mapping surface in accordance with said obtained placement information. As to 
the amended claim limitation of "automatically setting one mapping mode out of a 
plurality of mapping mode," it is clear that the computer automatically setting the 
mapping surface upon a user selection of a geometric surface . Xiong clearly teaches 
automatically setting the mapping surface out of a plurality of geometric mapping 
surfaces by the computer through a user selection. Furthermore, Teo teaches selecting 
among plural surfaces . Teo teaches manipulating (adjusting the scale of the polyhedral 
surface, rotating the surface, re-positioning edges of the polyhedral surfaces) a polyhedral 
surface upon which the scene is to be projected, relative to the initial panoramic image, to 
form a desired surface, wherein the desired surface is distinct from the initial surface and 
modifying the initial panoramic image by mapping points on the desired surface to 
corresponding points on the initial surface to produce a modified panoramic image 
corresponding to projection of the scene onto the desired surface. Teo therefore teaches 
automatically setting a mapping mode by the computer and selecting among plural 
surfaces . One having the ordinary skill in the art would have been motivated to do this 
because setting a mapping mode out from a plurality of mapping mode serves for the 
purpose of eliminating visible artifacts (Teo column 10). 
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Conclusion 

13. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated fi'om the mailing date of the 
advisory action. In no event, however, will the statutory period for reply expire later than 
SIX MONTHS fi-om the date of this final action. 

14. Any inquiry concerning this communication or earlier communications fi'om the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605- 
1213. The examiner can normally be reached on 8:00 AM - 4:30 PM, 

If attempts to reach the examiner by telephone are unsuccessfiil, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers for 
the organization where this application or proceeding is assigned are (703) 308-6606 for 
regular communications and (703) 308-6606 for After Final communications. 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 395- 
3900. 



jcw 

February 3, 2004 
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